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ABSTRACT 

Pavlov is a Programming By Demonstration (I'BD) system 
that allows animated interfaces to he created without 
programming. Using a drawing editor and a clock, 
designers specify the behavior of a target interface by 
demonstrating stimuli (end-user actions or time) and the 
(time-stamped) graphical transformations that should be 
executed in response. This stimulus-response model allows 
interaction and animation to be defined in a uniform 
manner, and it allows for the demonstration of interactive 
animation, i.e., game-like behaviors in which die end-user 
(player) controls the speed and direction of object 
movement. 

KEYWORDS 

End User Programming, (JIMS. Programming By 
DemonstraUon, Programming By Example, Animation 

INTRODUCTION 

A visitor to our planet might deduce Uiat most computer 
users have the necessary skills to quickly and easdy 
graphical user interfaces (GUIs). First, computer users 
know what they want: any user of today's popular 
applications is now quite capable of delivering a detailed 
(and passionate!) discussion on the suengths and flaws of 
computer interfaces. Second, most computer usirs have the 
mechanical skills required to demonstrate the appearance 
and behavior of an interface: anyone that has used a 
drawing editor knows how to draw objects widi a 
computer, click on them, and transform them. 

But today's development tools have yet to fully tap the 
potential of the computer user. Though interface builders 
like Visual Basic have significanUy decreased tie time and 
expertise necessary to build starulard interfaces, the 
development of more graphical, animated interlaces is still 
mostly performed by skilled programmers. This time- 
consuming and costly development is particularly un- 

Pcrmission to make digital/hard copies of all or pan of this material for 
personal or classroom us: is granted without fee provided that the copies 
are not nude or distributed for profit or commercial advantnge, the copy- 
right notice, the title of the publication and its dale appear, ind notice is 
given lhal copyright is by permission of the ACM, Inc. To :opy otherwise, 
to republish, to post on servers or to redistribute lo lists, requires specific 
permission and/or fee. 
CHI 96 Vancouver, BC Canada 
• 1996 ACM 0-89791-777-4/96/04.. S3. 50 



fortunate given die sxploding demand for computer games, 
interactive entertainment, and animation in even 
"standard" interfaces, and the recognized importance of 
more end-user participation in designing interfaces. There 
has been some [rogrcss: an entire book has beea 
published describing research systems that allow 
interfaces to be created or extended by demonstration 
rather than programming [2] ; commercially, tools like 
Macromedia's Director allow non- programmers to develop 
animation and sum; interaction. 

But none of these systems cohesively combine interactive 
techniques for specifying end-user interaction, graphical 
transformation, an 1 timing, the three primary ingredients 
of an animated interface. Director is powerful for 
specifying transformation and timing, but designing 
simple interaction requires some programming, and more 
complex interaction requires an expert. The Programming 
By Demonstration (PBD) systems in [2] present powerful 
techniques for specifying transformation and some 
interaction, but dn not provide the timing mechanisms 
necessary for animation. 

This paper presents Pavlov, a GUI development system 
based on the stimulus-response model. Stimulus-response 
provides a cohesi\e model for demonstrating interaction, 
transformation, an 3 liming. The model seeks to minimize 
the cognitive dissonance between concept and design by 
aJlowing designers to demonstrate die behavior of an 
interfuce exacdy as drey think of it: "When I do A. B 
occurs", or "two st conds after the start of the program, this 
animation begin.' ." Beginning with a blank target 
interface, tabula rasa if you will, die designer uses a 
drawing too! to draw the interface, then uses the same tool 
and a clock to demonstrate .stimulus-response pairs. In 
essence, the designer teaches the system in a way that is 
intuitive to humans: 

The basic ph /siological function of the cerebral 
hemisphere throughout the subsequent individual life 
consists in a censtant addition of numberless signaling 
conditioned stii tuli to the limited number of in-born 
unconditional stimuli, in olher words, in constanUy 
supplementing l ie unconditioned reflexes by conditioned 

Ivan Pcirovich Pavlcv 



252 



Sent by: SIEMENS TTB 



APRIL 13-18 



06/18/02 9:53; j£i£aa_#333;Page 3 




e I. Pavlov Development of a Driving Simulator 



^sHjnuius-response model was first used in the 
t's'DEMO system [13] to demonstrate non-animated 
: behaviors. The model has been exter.ded in 
';Sd that an interface can be taught about time, 
c activity, and the inherent direction of some 
/.These extensions allow animation as well as 
j^on to be designed within the stimulus-response 
"fork. It is this intersection between animation and 
J, not animation per se, that distinguishes Pavlov 
> systems. Because of it, design<jrs can 
rate most behaviors that combine interaction and 
, including game-like behaviors in which the 
layer) controls the speed and direction of object 



IP* 



NG SIMULATOR EXAMPLE 

ying simulator, the top car begins tnovin? when 
>|rom begins, follows a pre-defined path aro,ind the 
[fen stops near its starting point. The bottom car 
^W*oving only when the driver rotates the 
gOtor". Its speed and direction is controllea by the 
(the end-user) manipulating the accelerator and the 
iyheel. 

shows the Pavlov environment during 
pient of the driving simulator (also see die Video 
p the CHI 96 Video Program). The basic tcols are 
[Wing editor in the top-right corner, the clock 
jghf), and the Development Mode Palette {lower- 
ie designer uses the development modes to inform 
a whether s/he is just drawing the interface 
jjBQde), demonstrating an end-user or time stimulus 
s mode), demonstrating how (he system should 
j:\tp.a stimulus (Response or Real-Time Response 
~f testing an interface (Test mode). The designer 
*'ock to demonstrate when an operation should be 
^ jslng the top At: box), or if an operation should 
P^j. periodically (the middle Every: box). The 
n the clock, labeled Record Time Stirr.ulus, 



Figure 2. The Pavlov Editor 



allows the designer to specify that the following responses 
should be triggered by rime. 

Another important part of the environment is the editor, 
shown in Figure 2 This editor displays a textual 
description of the ini;rface being designed. It serves to 
provide feedback to a designer as demonstrations are 
performed, and it allows the designer to modify the 
behaviors "learned" b) the system when necessary. Details 
of this editor are provi led later in the paper. 

The first task in creadug the Drawing Simulator is to draw 
the two cars, the roatl, the accelerator, and the steering 
wheel. To do so, the d;signcr selects Draw mode from the 
Development Mode Pilette, and makes use of the graphic 
primitives and group big mechanisms available in the 
drawing palette. He iJso provides names to the drawn 
objects for later reference. 

Next, the designer be.jins specifying the behavior of the 
interface. Because s/ht wants the two objects shaped like 
cars to move as cars do, s/he selects each, chooses Object I 
Set Direction from the menu and enters an angle that 
defines the direction attribute of the particular car. In the 
simulator, both cars initially face straight ahead on the x- 
axis, so the designer sets the angle to 0. A vector 
emanating from its ceiter appears on each car to signify 
that the car will only move forward and backward in 
relation to its direction, and must be routed to change 
direction. The vector does not appear during execution of 
the target interface. 

The designer is now leady to demonsorale the stimulus- 
response behavior of the top car. In this case, the stimulus 
is time: at lime 0 (th: beginning of execution) the car 
should begin moving. ' "o demonstrate, the designer selects 
Stimulus mode, sets thj clock At: box to 0, and clicks on 
the Record Time Stimi lus button. The system reports that 
a time stimulus has teen recorded, and automatically 
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switches to Response mode. For this behavior, the designer 
wants to demonstrate a special kind of response, a real- 
time response, so that mode is selected. The designer then 
selects the move icon in the drawing palette and drag* the 
car around the track. As s/he drags the car, it doesn't ever 
move diagonally, but instead moves forward towards its 
nose, and rotates its base (turns) in order to follow the 
mouse around the track. After the designer release* the 
mouse button, s/he sees from the editor that the systerr, has 
recorded a series of discrete, time-stamped responses, 
made up of alternating MoveForward and Rotate 



Next, the designer enters Test mode to visually test the 
demonstrated behavior. Immediately, the top car begins 
moving and follows the path that was demonstrated. The 
designer knows s/he could edit the recorded responses in 
the editor to modify the path, but for now s/he is satisfied 
with the behavior of the lop car. 

The designer then turns his attention to the bottom car. 
The bottom car's behavior is not triggered by time, bit by 
an end-user action. Thus, instead of demonstrating a rime 
stimulus, the designer plays the role of the end-user and 
demonstrates an action. After entering Stimulus mode, 
s/he selects the Rotate icon in the drawing palette, presses 
the left-mouse button on the rectangle denoting the 
accelerator, and rotates it clockwise some amount, say 
-0.36 radians. The system reports (hat a stimulus was 
recorded and automatically switches the development 
mode to Response. The system also records an implicit 
stimulus-response descriptor mapping the physical action 
used to the higher-level operation: 

(1) On Accelerator .LefiDrag --> Accelerator Jlotate 

At this point, the designer needs to demonstrate that the 
rotation of the accelerator should cause a response of 
accelerating the movement of the bottom car. First, .i/he 
enters 1 in the Every: box of the clock. S/he knows that 
this will cause the upcoming demonstrated response to be 
executed periodically every time frame in the target 
interface. Next, s/he moves the car some amount, say 17 
units. Because the car is a "directed" object, the cir's 
movement is restricted: it can only be moved on the vector 
defined by its direction arrow (note that in Real-Time 
Response mode this vector is allowed to change). Ibis 
restriction is as the designer desired: in response to the 
rotation of the accelerator, s/he wants the car to move 
forward, not change direction. The system reports the 
recorded stimulus-response descriptor containing a 
proportional constant (-47.22 = 1 11-36) 

(2) On Accelerator.Rotate(s 1 )--> 

BoUomC.ar.MoveForward(-47.22*sl) At 0 Every 1 
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Next, the designer enters Test mode to check his work. 
The top car immediately begins its path. The designer, 
playing the role of the snd-user, rotates the accelerator; . 
The bottom car begins moving, and continues to move 
even after the designer releases the mouse from the 
accelerator. As the car leaves the right side of the screen, it 
reappears on the left. The designer again experiments with 
rotating the acceleratoi and notices that rotating it .'ffff 
clockwise speeds up the car, while rotating it counteri' * 
clockwise slows it down. ; 

The designer is nearly satisfied but thinks the accelerator is 
a little sensitive. He enters the editor (see Figure 2) and, 
selects "Accelerator" as the stimulus object. The stimulus; 
"Rotate(sl)" appears in he box labeled stimulus, and J J 
single response appears ;n the first cell of the score mis- 
labeled BottomCar. The response box below the score £ 
contains the response "MoveForward". Its parameter, as ibf! 
descriptor (2) above, is -47.22*sl. The period box contains;! 
"1". To reduce how mucl the car speeds up in response $ 3 
the rotate, the designer changes the proportional facti^-l 
from -47 to -30.0. (altei natively, the period could have J 
been increased). When sfce re-enters Test .mode. s/b^jr$ 
satisfied to see that the accelerator is indeed less sensitive!!^ 

"•pi 

The next task is to specify the behavior of the sieeri«|:|| 
wheel so the end- user can control the direction of U)£;|f 
bottom car. The designer inters Stimulus mode and rotaiM ji|j 
the steering wheel. Then, in Response mode, s/he sets Ut'l^S 
Every: box in the clock io 1, and rotates the bottom cai X 
The following descriptor is recorded: " 51 

(3) On Wheel.Rotate(sl) -> J r\ 
BottomCarJl >tate(0.25*sl) At 0 Every I };|| 

To test this new behavior , the designer once again eniei$$ 
Test mode. The top car immediately begins its path, ftfelf 
designer rotates the accelerator to get the bottom dgjjff 
moving, then releases tie accelerator and rotates rhpfe 
steering wheel to control i s direction. He is pleased to nc^m 
that s/he was correct io set the Every: box beft&j^fl 
demonstrating the rotation of the bottom car: just Kke'pf^ 
real one, the car continu :s to turn if the steering whiE^|| 
remains rotated from its oiiginal setting. -j.-fb 

The designer continues o test the interface, and satifc? 
realizes that if s/he relates the accelerator rounrj&jp 
clockwise past its origin, I be car begins moving backwar$*l 
To alleviate this problem, s/he uses the editor to delete ihfe^ 
previously recorded accelerator behavior, and then fe*| 
demonstrates it. First, in Draw mode, s/he sets the top^lejgj 
point of the accelerator oi the left edge of the enclosi^ 
rectangle and chooses Conditions I Generate from jSp" 
menu. Then s/he demonstrates the stimulus of rotating ''p|p 

accelerator. A dialog apt ears listing a, set of grapbi||]p|| 
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». conditions relating the stimulus object to other objects in 
phe interface. The designer selects the condition 

^Accelerator.WithinfEnclosingRectangle)", and ihe system 
iifecords a modified version of the originally recorded 

Romulus-response descriptor (1). 

fi(4) On Accelerator.LeftDrag --> Accelerator .Rotate 
When Accelerator. Within(EnclosingRectangle) 



r proceeds to demonstrate the response of 
^-moving the car forward, as s/he did in the first iteration. 
§' Afterwards, s/he re-enters Test mode, and is satisfied to see 
|jbat s/he (playing the role of the end-user) is restricted 
tprom rotating the accelerator outside its enclosing 
J^rectangle. Since the top-left point of the accelerator begins 
||on the left edge of the enclosing rectangle, there is no way 
tp rotate it counter-clockwise past its origin, so the car 
p-annotmove backward 

UpTHE STIMULUS-RESPONSE MODEL 
|.Tbe driving simulator example illustrates many features of 
\ the stimulus-response model, including the extensions that 
' allow interactive animation to be defined. This section 
^Kg describes the model in more general terms in order io 1) 
rexplain the inferences made by the system in the example, 
H and 2) bridge the gap between the specific and the general, 
jg<;i.e., persuade the reader that Pavlov is useful for designing 
; all kinds of interfaces, not just driving simulators. 

if An interface is viewed as a stimulus-response machine, 
if Stimuli are either physical actions (e.g., drag the mouse 
£*hile pressing the left-buaon), higher level operations, or 
¥ time. The interface responds to stimuli by executing a set 
£6f time-stamped operations. Operations either create, 
iyl-iransfonn, or delete objects. The set of operations includes 
j|p the primitives found in most drawing editors and one 
additional primitive, move forward. This additional 
^primitive allows an object to be moved while constrained 
hto the vector defined by its direction attribute. Together, 
b the operations offer the base functionality necessary to 
ipv demonstrate nearly all animated interface behaviors. 

The Semantics of Stimulus-Response 

p| ; The challenge of a stimulus-response development system 
fe,is to provide clear syntax and semantics for how the 
jk" , designer uses the set of physical actions and operations to 
; demonstrate the behavior of the target interface. The 
, : "syntax" of Pavlov is straight forward: the designer 
c changes development modes to inform me system whether 
his intent is to draw, demonstrate a stimulus or respmse, 
or test the interface. 

Providing clear semantics is a more challenging problem: 
: the goal is for the system to always record a stimulus- 
" " f response descriptor that perfectly matches the intent of the 



designer's demonstration (this is the primary challenge of 
all PBD systems). Pavlov uses an explanation-based 
learning approach: froci a single demonstration of a 
stimulus-response pair, tie system uses domain knowledge 
and the information provided by the demonstration to 
record as reasonable a stimulus-response descriptor as 
possible. If necessary, th j designer can then use Pavlov's 
powerful editing facilities to modify the descriptor. 

This simplistic strategy differs from other systems that 
allow a designer to refit ie behavior descriptions through 
multiple demonstrations. Such an empirical-based 
learning approach allows more complex behavior to be 
specified, but complicates the semantics of the system. 

The Semantics of a Stimulus Demonstration 

In Stimulus mode, the designer demonstrates the 
operations the end-user cm perform in the target interface. 
When the designer performs an operation in this mode, 
Pavlov records 1) a stimulus-response pair mapping the 
physical action used to the operation that was executed, 
and 2) the first part of a second stimulus-response pair that 
will eventually map the operation to one or more 
operations demonstrated ;is the response. 

Mapping the physical ac ion to me operation is important 
because the drawing pal;tte used during development to 
demonstrate operations does not appear when die target 
interface is executed (Te st mode). In Test mode, the end- 
user can only use the operations that the designer has 
explicitly demonstrated is stimuli, and can only access 
those operations using tie physical action (mouse button, 
auxiliary key) used in Uk demonstration. For example, in 
the driving simulator tie end-user can only rotate the 
accelerator by dragging the mouse with the left-mouse 
button down, because that is how it was demonstrated. The 
designer cannot manipulate the car directly in any manner, 
because no such stimulus was demonstrated. This positive 
example method of specifying the functionality of the 
system is in contrast to the scheme of [9] in which the 
designer "freezes" the ob ects that cannot be manipulated. 

The second recording made from a Stimulus 
demonstration records the high-level operation 
demonstrated as a stimulus (e.g., AcceleratorJRotate). It is 
the execution of this high-level stimulus that will trigger 
the execution of the resionses demonstrated in Response 
mode. 

The only complication to die semantics of a stimulus 
demonstration is that i designer may demonstrate a 
stimulus on a representative object. At run-time, the same 
stimulus applied in any riember of the set represented will 
trigger the demonstrated response. Dynamically allocated 
objects, which can be sjecified by creating an object in 
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stimulus or response mode (see [13]), are by default 
marked as representative objects. Pavlov also allows the 
designer to designate behavior groups, and marks each 
element as representative of the group (a similar approach 
is used in f 12)). 

The Semantics of a Response Demonstration 
When the designer demonstrates an operation R on object 
O in response mode, the system connects a response of the 
form "O.R (r^J when C" to the previously 
demonstrated stimulus, where each r, is a function of zero 
or more stimulus parameters, and C is an optional context 
for when R should be executed. 

The simplest semantic rule is to execute the demonsiraied 
response each time the demonstrated stimulus occurs :n the 
target interface. However, such a simple rule would 
preclude the designer from demonstrating the context for 
when an operation should be executed; semantics su:h as 
"execute R is response to stimulus S only wher. the 
environment is in state s" could not be demonstrated. 

By setting a toggle, the Pavlov designer explicidy states if 
context should be taken into account. If it is, the designer 
configures the interface into the desired context (or the 
negation of the desired context) prior to a resronse 
demonstration. After the demonstration, the system nins a 
set of tests to identify graphical conditions dcscribinj; the 
state of the interface. The designer is allowed to select one 
or more of these conditions and combine them with logical 
operators to define the context for when a response should 
be executed. 

In the driving simulator, a context was defined on the 
description mapping the physical stimulus 
"Accelmtor.LeftDrag" and the response 
"Accelerator.Rotate". A context could also be demonstrated 
so that the bottom car doesn't run into the top one: the 
designer demonstrates the stimulus of rotating the 
accelerator, tells the system to identify context, then 
demonstrates a response of moving the bottom car so that 
it intersects the top car. When the system identifies 
BottomCar.Intersects(TopCar) amongst other conditions, 
the designer selects it and negates it, and the following 
behavior is recorded: 

(5) On Accelerator .Roiate(sl)-> 

BoUomCar.MoveForward(47.22*sl) At 0 Every 1 
When Not (BotlomCar.IntersecisaopCar)) 

la general, mere are many true conditions concerning the 
state of the interface. To reduce the number of conditons 
listed for the designer, the system only identifies those 
conditions relating the response object and all other objects 
in the interface. Because the stimulus object has also teen 
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signified as important, relationships found concerning .itilj 
and the response object are shown at the top of the list'bf^f 
found conditions. When necessary, the designer can use?l| 
the editor to specify i. condition not identified by the^f 
system. 

A second complication io the response semantics is similar! 
to that discussed in the stimulus section: if 'fo£g 
demonstrated response object is representative of a set, thei 
response is applied to the entire set during execution (, 
subset, as defined by a context conditional {13]). 

A third complication tc the response semantics concenisj| 
determining the response parameters. The sinuilesiS? 
solution is, of course, tc execute R during execution wifitft 
the same parameters as in the demonstration. This is d 
best solution when the corresponding stimulus has j 
parameters (e.g., the st mulus is a button click and t 
response is a move(x=5,j'=7)). However, for stimuli the 
have parameters, it is often the case that the reactic 
proportional, i.e., the response parameters) 
proportional to the piirameters of the stimulus, fjpi 
instance, the car in the Driving Simulator is rotaied/fjotl 
amount proportional to the amount the steering wheel isi 
rotated. Thus, when such a stimulus-response ;.jT 
demonstrated, Pavlov irfers proportional constants Cj';j 
rj/si. that relate each of the stimulus and respon&l 
parameters. When the {tinjulus S(si,s 2 ,...) occurs durij 
execution, the response B (s,.C,. s 2 . C 2 ) is executed. 

The formula for R illustrates that the system infers the fS 
parameter of the stimulus to be related l 
parameter of the response, the second to the second, and; 
on. Tlie basis of this nference is that most inte. 
operations either have a . angle parameter or they have if. 
parameters denoting x and y coordinates, so in practice {ft 
corresponding stimulus and response are often related; < 
with conditionals, the alitor can be used to modify. ?| 
response formulas record xi. 

EXTENSIONS FOR ANIMATION 

Systems for demonstrating animation have existed for o 
twenty-five years [1]. Pavlov's contribution 
integration of animation demonstration wilh the s 
response model for defining interaction. 

An animation path can le demonstrated with a real-til 
response demonstration, with a series of time-s 
response demonstrations (the editor can be used for..;jj 
bctweening). or with a periodic response. Like any c 
response, an animation p ith can be triggered by any lc 
of end-user or time stimulus. 

When a designer demonstrates the transformation of Jt 
object in real-time response mode, die system recor 
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s of time-stamped operations. Because operations are 
i instead of picture frames (as in Director), the 
J path is not constrained to a particular starting 
r object. Thus, reuse is facilitated. The mechuiism 
> slightly more general than in systems such as 
•'r because any operation, not just move, cm be 
istrated in real-time. 

ption of Direction 

important contribution of Pavlov is that a designer can 
* mstrate animation in which the end-user not only 
i movement, but accelerates it and changes its 
n. Though such behavior is the primary activity in 
ime-like applications, there has been litde research 
i area, and commerciaJ systems such as Director 
e extensive programming to develop this part of an 
ation. 

- struggling with how to allow game-like behav:or to 
KMistrated, the following observations were made: In 
! games, one input control (e.g., steering wheel) is 
control direction, and a different control 
or) to control speed. Also, many objects do not 
n an arbitrary manner, but are restricted to moving 
d and backward, and must rotate their base to turn. 

e observations it became clear that the standard 
x,y) operation in Pavlov's drawing editor i«. not 
t for the demonstration of movement because it 
s both a distance and an absolute direction. 

" r §plve the problem, a notion of direcdon was addid to 
stimulus-response model. Designers can set a current 
ion attribute for an object that is displayed during 
lopment The direction attribute makes it possible for 
mer to demonstrate a MoveForward(d) open don. 
operation causes an object to move forwarc (or 
1, if d<0) io the direction it is facing. Thus, it is 
r suited for the demonstration of acceleration 
Move(x,y). 



! notion of a periodic response is also useful in 
lsirating acceleration. In the driving simulator, when 
i end-user rotates the accelerator, the car should legin 
5 and continue to move, even after the end user 
the mouse. In Pavlov, the designer explicidy 
es continuous movement by setting the Every, box 
the clock before the demonstration of a forward 
menL In essence, when a "MoveForward (d) Every t" 
: demonstrated, the system infers that the object should 
~ve forward at a speed of d/t 

e following run-time rule follows from these scmartics: 
execution of successive periodic MoveFor*ard 
rations on the same object results not in two alternating 



and possibly opposite actions, but in a single action 
combining the magnituces of the operations. For example, 
in the driving simulator, when the car is already moving 
at 4 units/frame and the end-user rotates the accelerator 
again, say back towards the origin, it causes a response of 
MoveForward (-1) uniu/frame. This second operation is 
combined with the existing one so that the car slows down 
to 4-1=3 units/frame, instead of alternating between 
moving forward 4 units, and backward 1 unit. 

The system only uses these semantics for periodic Move- 
Forward operations. For other operations, successive 
periodic responses will execute in tandem. Thus, using two 
periodic, regular Move demonstrations, the designer can 
demonstrate that an object move back and forth, such as in 
an animated move icon. 

An alternative method of demonstrating acceleration has 
also heen added to (he Pavlov environment. After 
demonstrating a stinulus that should cause the 
acceleration, the designe r enters real time response mode 
and moves an object forward at the desired speed. 
Generally, a real-time response is used to demonstrate a 
fixed animation path as a response to a simple button-click 
or time. When a real-time Move Forward is demonstrated 
as a response to a transformational stimulus (one with 
parameters), the system ioes uot record a series of discrete 
time-stamped operations as usual, but instead records a 
single periodic operation. The distance parameter is 
computed by dividing the total distance of the 
demonstrated movement by the time of the movement (d/t), 
and the period is set to 1 (ms). 

The advantage of this scheme is that the designer truly 
demonstrates the speed of the movement; the disadvantage 
is it complicates the stmantics of the system. A more 
thorough analysis will bn provided after more feedback is 
gathered from users. 

THE PAVLOV EDITOR 

An important aspect of i PBD system is how a designer 
edits the behaviors inferred by the system. Pavlov's editor 
borrows from Director hy providing a time-line view of 
activity (a score). Htwever. because interaction is 
emphasized, Pavlov pro\ides multiple timelines: one for 
the events that occur without an end-user stimulus, and 
one for the events triggered by each end-user stimulus that 
was demonstrated. This method of organizing events by 
stimulus significantiy eaies the editing task compared to 
the single score editors fcund in most animation systems. 

Pavlov's editor, shown in Figure 2, can be viewed 
simultaneously with the main development window. In 
order to view the operators that occur in response to a 
particular stimulus, the iesigner selects an object and a 
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particular stimulus in the top-left list boxes. To view the 
operations that occur without an end-user stimulus the 
designer selects the "Time Stimulus" check box to the 
right of the stimulus list 

The objects that respond to the listed stimulus are shown in 
the rows of the score. The designer can select a particular 
response in a cell, and the inferred response parameters 
appear in the edit boxes labeled Rl and R2. Any 
expression consisting of constants, stimuli parameters, and 
system-supplied object attribute functions may be entered 
as a response parameter. In essence, editing behavior 
formulas is very similar to entering a formula in a 
spreadsheet 

IMPLEMENTATION 

At the beginning of execution (Test mode), the time 
stimulated operations defined in the target interface are 
placed in the execution list with their respective time 
stamps. For each end-user stimulus that occurs, the sr 
processor traverses the selected object's stimulus-response 
list to find the response operations associated with the 
stimulus. These responses are copied into the execution list 
with a time-stamp of t + t, . where t is the time the 
stimulus occurred (the current time), and t, is the recorded 
time-stamp of the response (which is relative to the 
stimulus). When the system is not processing end-user 
stimuli in this manner, the execution list is traversed and 
all operations whose time-stamp is less than the curr;nt 
time are executed. A non-periodic operation is removed 
from the execution list immediately after execution; a 
periodic operation is left in the list, with its time-stamp 
incremented by the size of its periodic interval. In either 
case, the executed operation is sent as a stimulus to the sr 
processor, so a chaining of events can occur. 

Though the scheme does not guarantee that operations will 
be executed before or on their time-stamp, in practice it 
provides visually acceptable performance even for 
mterfaces with lots of interaction and concurrent 
animation (Pavlov runs on a 486 PC). 

RELATED RESEARCH 

Rehearsal World [5] and Peridot [7] were early PBD 
systems that inspired the stimulus-response framewoik. 
The first systems to allow direct graphical demonstration 
of a full range of stimuli and responses were DEMO [13] 
and Marquise [8]. DEMO introduced the stimulus- 
response model and a technique for demonstrating 
dynamically created objects, while Marquise focused on 
the demonstration of graphical editors, including those 
with palettes and modes. 

OEMO II [3] and [4J are stimulus-response systems tint 
allow the designer to perform multiple demonsuations of 
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the same behavior to refine the system's inferences Gnu 
uses multiple examples to make sophisticated mreren2| 
concerning response parimeters-- inferred formulas mMf 
depend on attributes 3 f arbitrary objects as welhM 
stimulus parameter values. DEMO 11 uses multipll 
examples to refine inferences concerning the context i 
when a response should b ; executed. 

Pavlov is the first stim lus-response system to focus Si 
animation, though there :i re a few PBD systems not baj#l 
on stimulus-response that allow some animation tbvrjgS 
demonstrated: KidSim [2] and Agent Sheets [10] i 
graphical rewrite rules tr allow designers to demonstr, 
the context for when an operation should be executed, 1 
systems are powerful for creating non-interactiy| 
simulations, but the rewrite-rule method of defihi%s 
context is not integrated vith a method of specifying ejuSf 
user stimuli, so interactive simulations cannot be desighftdf 
w.thout coding; Dance pi] allows the demonstratidoW^ 
animation for the purple of program visualizatk 
Chimera [6] and LEMM1SG [9] allow interface behavL, 
to be specified with multiple demonstrations 
constraints, but do not nvcr time-based animation^ 
acceleration. 

Director is representative of the commercial animat ^ 
systems that provide fa duties for both animation aig 
interaction design. These systems allow animation ti^jjj 
designed quickly and casil / using a combination of fi "" 
by-frame animation, in-betweening, and real-umjg 
recording. These systems jlso allow sound and vidtxrip-£if 
linked into presentations, uid provide a range of feato* 3 ** 
for creating special effects such as slow-in/sloW 
motion blur, and squash a vd slretch. 

Though powerful for defin ng animation, these systems! 
not provide a PBD metiicd of defining interaction,'^ 
systems all allow button-click triggered animation td j 
defined in a relatively sinple manner. However, riwts 
complex stimulus-response behaviors, such as the steenn| 
wheel and accelerator controlled animation in the driyf" 
simulator, require expert-le /el programming. 

A second difficulty in defiling interaction with a. 

systems is that they are based on a single-score editor! ? 
the animation sequences of an application are shown ori-| 
single time-line. Though su^i a score is sufficient for hC 
interactive animation (whici was its original purposeKitttf 
loo unstructured for applica ions with interactive as weUlsjf 
time-stimulated animation. Like the programs wrint 
before the advent of structured programming (silt,* 
procedures), the designer is forced to program control, $g*| 
where one animation ends ;ind another begins, using $ 
statements. For complex applications with lotS ;' 

movement and interaction, the result is a spaghetti i 
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Multiple score scheme in the Pavlov editor alleviates 
iblem, and allows the designer to edit the different 
e behaviors and animation sequences separately. 

AT10NS 

r practical limitation of Pavlov is that interfaces 
ith it cannot be connected to application code. In 
t version, designers will be able to make this 
: ~n by 1) demonstrating a function call sis a 
or response, and 2) calling an application 
within a response or conditional formula, 
single demonstration scheme might also be 
J a limitation: more behaviors could be inferred 
uttiple demonstration inference engine, such as [4], 
htegrated. Before doing so, however, we want to study 
the additional inferred operations justify the 
complexity that would be added to the 
an. A third limitation is that acceleration csn be 
strated for an object thai has no pre-defined path, 
>t be demonstrated for an object that must stay on 
uh. In this regard, we are exploring both tin: use 
metric functions to model some animation paths, 
jhe use of "conductor" objects with special properties 



contributes a cohesive model for demonstnting 
i and interaction, and innovative techniques for 
istrating interfaces in which the end-user controls 
: speed and direction of animation paths. Using 
chniques, interfaces like the driving simulator can 
i in less than fifteen minutes. 

relopmcnt is by no means restricted to driving 
ulators or similar applications. A number of other 
have also been developed, including a wide 
of games, a diagram editor with animated icons, 
an educational solar system program. We attribute 
^general usefulness of Pavlov to the generality of the 
uius-response model, and its powerful multiple-score- 
editing facility. 

increasing the range of PBD, the slirrulus- 
"ponse model provides a very intuitive method for 
^fining interfaces. A usability test was performed v/ith a 
" ber of non- technical designers. Subjects were given a 
ual describing the stimulus-response model, and then 
1 asked to design two interfaces equal in complexity to 
driving simulator. Seven of ten were able to create the 
'^ferfaces within an hour, the three others complcud the 
;tj$ks after asking a few questions (specific questions 
:erning how to complete the tasks were not allowed). 
We were extremely encouraged by the results, as well as 
ihe enthusiasm the subjects expressed for exploring once 



the formal tests were completed (though we could get 
none to salivate!). 
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Pygmalion was an early attempt to change the process of programming from one ■HaflUHBDBl 
in which algorithms are described abstractly in a programming language to one 
in which they are demonstrated concretely to the machine. It introduced two new 
concepts: programming by demonstration and icons. Icons have become widely 
accepted. Programming by demonstration is still waiting for a practical system 
to show its validity. Pygmalion was completed in 1975. This chapter is a com- 
mentary on that work. 



TraditionaUy people instruct computers by sitting down with a pad of paper (or a ftu^r^bmp'uler ! ' ?M 
window for a text editor) and writing a sequence of statements in some pro- 1 Ci^UlifiSltl^4u¥& 
gramming language. They then compile the statements, link the resulting ma- 
chine code with other run-time routines, and attempt to execute iL There are al- 
most always errors. At this point programmers enter the debug-edit-recompile 
loop. They track down bugs using whatever means are available, edit the source 
text of their programs, recompile it, relink it, and reexecute it. There are almost 
always more errors. In fact, fixing a bug often introduces new bugs. This debug- 



edit-recompile loop can consume the majority of time spent on a program. And 
no one claims anymore that all bugs can be removed from any large program 

Writing static language statements interspersed with compile-run-debug-edit 
periods is obviously a poor way to communicate. Suppose two humans tried to 
interact this way! Specifically, it is poor because it is: 

(a) Abstract 

The programmer must mentally construct a model of the state of the machine 
when the program will execute, and then write statements dealing with that 
imagined state. In practice, for a program of any complexity this is impossible. 
People cannot handle as much complexity as computers can. A typical symptom 
of this is boundary errors: an unanticipated value is used in an operation, result- 
ing in an error. Dividing by zero is a classic example. We must find a way to al- 
low programmers to work with concrete values without sacrificing generality. 

(b) Non-interactive 

The debug-edit-recompile loop takes a lot of time per bug fix, particularly as 
programs get large. This low productivity has led programmers to seek faster 
machines and compilers. But that misses the point Instead of speeding up the 
debug-edit-recompile loop, programmers should eliminate it! A few program- 
ming languages such as Lisp and Smalltalk are in fact interactive. One can type 
in an expression and immediately get it evaluated and see the result Such lan- 
guages are more productive than non-interactive languages. 

So why aren't interactive languages more widely used? Most are high level lan- 
guages that are not as efficient as low level ones such as C and FORTRAN. In 
their search of a competitive advantage, software developers want their pro- 
grams to execute as fast as possible, even at the expense of a longer develop- 
ment time. But this situation may be changing. Computers are now fast enough 
that interactive languages are practical in some cases. Several commercial appli- 
cations have been written in Lisp, and IBM now recommends Smalltalk for de- 
veloping applications under its OS/2 operating system. Even some versions of C 
now contain an interpreter for quick development and debugging. 

(c) "Fregean" 

The most articulate representation for a program requires the least translation be- 
tween the internal representation in the mind and the external representation in 



the medium. Aaron Sloman [Slomao 71] distinguishes 
sentations: "analogical" and "Fregean" (after Gottlob 
predicate calculus). Analogical 
things they describe; Fregean representations have no 
the 



two kinds of data repre- 




Analogical 



structure to the 
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One of the advantages of analogical representations over Fregean ones is that 
structures and actions in a context using analogical representations (a 
"metaphorical" context) have a functional similarity to structures and actions in 



the context being modeled. One can discover relationships in a metaphorical 
context that are valid in the other context For example, there are five items in 
array A; triangles have three sides; bicycles have two wheels; Texas is in the 
south. Analogical representations suggest operations to try, and it is likely that 
operations applied to analogical representations would be legal in the other con- 
text, and vice versa. This is the philosophical basis for the design of the Xerox 
Star and, ultimately, Apple Macintosh "desktop" user interfaces; they are a 
metaphor for the physical office [Smith 82]. Being able to put documents in 
folders in a physical office suggests that one ought to be able to put document 
icons in folder icons in the computer "desktop," and in fact one can. 

Jerome Bnmer. in his pioneering work on education [Bnmer 66], identified three 
ways of thinking, or "mentalities": 

• enactive - in which learning is accomplished by doing. A baby learns what a 
rattle is by shaking it. A child teams to ride a bicycle by riding one. 

• iconic - in which learning and thinking utilize pictures. A child learns what 
a horse is by seeing one or a picture of one. 

• symbolic - in which teaming and thinking are by means of Fregean symbols. 
One of the main goals of education today is to teach people to think 
symbolically. 

But Bnmer argues that it is a mistake for schools to force children to abandon 
their enactive and iconic thinking skills when teaming the symbolic. All three 
mentalities are valuable at different times and can often be combined to solve 
problems. All three skills should be preserved. 

Jacques Hadamard, in a survey of mathematicians [Hadamard 45], found that 
many mathematicians and physicists think visually and reduce their thoughts to 
words only reluctantly to communicate with other people. Albert Einstein, for 
example, said that The words of the language, as they are written or spoken, do 
not seem to play any role in my mechanism of thought The psychical entities 
which seem to serve as elements in thought are certain signs and more or less 
clear images which can be 'voluntarily' reproduced and combined.... This com- 
binatory play seems to be the essential feature in productive thought— before 
there is any connection with logical construction in words or other kinds of signs 
which can be communicated to others.... The above mentioned elements are, in 
my case, of visual and some of muscular type. Conventional words or other 
signs have to be sought for laboriously only in a secondary stage, when the 



mentioned associative play is sufficiently established and can be reproduced at 
will." [Hadamaid 45, pp.142-3] 

Some things are difficult to represent analogically, such as "yesterday" or "a 
variable-length array." But for concepts for which they are appropriate, analogi- 
cal representations provide an intuitively natural paradigm for problem solving, 
especially for children, computer novices, and other ordinary people. And a 
convincing body of literature suggests that analogical representations, especially 
visual images, are a productive medium for creative thought [Araheim 71]. 
Words are Fregean. Yet words and rather primitive data structures are often the 
only tools available to programmers for solving problems on computers. This 
leads to a translation gap between the programmer's mental model of a subject 
and what the computer can accept / believe that misunderstanding the value of 
analogical representations is the reason that almost all so-called visual pro- 
gramming languages, such as Prograph, fail to provide an improvement in ex- 
pressivity over linear languages. Even in these visual languages, the representa- 
tion of data is Fregean. 

With the advent of object-oriented programming, the data structures available 
have become semantical! y richer. Yet most objects are still Fregean. A pro- 
grammer must still figure out bow to map, say, a fish into instance variables and 
methods, which have nothing structurally to do with fish. 

(d) "Blank canvas" syndrome 

Pablo Picasso said "the most awful thing for a painter is the white canvas." One 
of bis most memorable paintings, "The Studio at Cannes," is of his own studio. 
Its walls and ceiling are covered with paintings. Priceless works of art are jum- 
bled together everywhere. But right in the middle of the room stands an easel 
holding a blank canvas. This is a fitting image of the problem facing all creative 
people: how to get started. A blank coding pad is as much a barrier to program- 
ming creativity as a blank canvas is to a painter. 

What's needed is a lightweight, non-threatening medium like the back of a nap- 
kin, wherein one can sketch and play with ideas. The computer program de- 
scribed in the remainder of this chapter, Pygmalion, was an attempt to provide 
such a medium. Pygmalion was an attempt to allow people to use their enactive 
and iconic mentalities along with the symbolic in solving problems. 



In Roman mythology, Pygmalion was a sculptor of extraordinary skill. At the 
peak of his powers, he determined to create a masterwork, a statue, of a woman 
so perfect that it would live. The statue, which he named Galatea, was indeed 
beautiful, but alas it remained just stone. Heartbroken, he prayed to the gods. 
Venus, impressed with his work and passion, took pity on him and brought me 
statue to life. (They could do that back then.) 

This urge to create something living is common among artists. Michelangelo is 
said to have struck with his mallet the knee of perhaps the most beautiful statue 
ever made, the Pieta, when it would not speak to him. And then there's the story 
of Frankenstein. Artists have consistently reported an exhilaration during the act 
of creation, followed by depression when the work is completed. Tor it is then 
that the painter realizes that it is only a picture he is painting. Until then he had 
almost dared to hope that the picture might spring to life." (Lucien Freud, in 
[Gombrich 60], p.94) This is also the lure of programming, except that unlike 
other forms of art, computer program s do "come to life" in a sense. 

The Pygmalion described in mis chapter is a computer program that was de- 
signed to stimulate creative thinking in people [Smith 75]. Its design was based 
on the observation that for some people blackboards (or whiteboards these days) 
provide significant aid to communication. If you put two scientists together in a 
room, there had better be a blackboard in it or they will have trouble communi- 
cating. If there is one, they will immediately go to it and begin sketching ideas. 
Their sketches often contribute as much to the conversation as their words and 
gestures. Why can't people communicate with computers in the same way? 

Pygmalion is a two-dimensional, visual programming environment implemented 
on an interactive computer with graphics display. (Although this work was com- 
pleted nearly two decades ago, I will describe it in the present tense for readabil- 
ity.) It is both a programming language and a medium for experimenting with 
ideas. Communication between human and computer is by means of visual enti- 
ties called "icons," subsuming the notions of variable, data structure, function 
and picture. Icons are sketched on the display screen. The heart of the system is 
an interactive "remembering" editor for icons, which both executes operations 
and records them for later reexecution. The display screen is viewed as a picture 
to be edited. Programming consists of creating a sequence of display images, the 



last of which contains the desired information. Display images arc modified by 
graphical editing operations. 

In the Pygmalion approach, a programmer sees and thinks about a program as a 
series of screen images or snapshots, like the frames of a movie. One starts with 
an image representing the initial state and transforms each image into the next 
by editing it to produce a new image. The programmer continues to edit until the 
desired picture appears. When one watches a program execute, it is similar to 
watching a movie. The difference is that, depending on the inputs, Pygmalion 
movies may change every time they are played. One feature that I didn't imple- 
ment but wish I had is the ability to play movies both backward and forward; 
then one could step a program backward from a bug to find where it went 
wrong. 

There are two key characteristics of the Pygmalion approach to programming. 

• It relies on editing an artifact rather than typing statements in a program- 
ming language. Editing has proven to be easy for people. Everyone who 
uses computers— over 100 million people today— can use text and graphics 
editors, but hardly anyone can "program." 

• The screen images always contain concrete examples of the program's data 
This eliminates an entire class of errors due to abstraction. 

Today we recognize these as being good user interface principles that most per- 
sonal computer applications heed. So one way to characterize Pygmalion is to 
say that it attempted to bring good user interface principles to the process of 
programming, not just to the result of prograrnming. 

Another early attempt to specify programming by editing was Gael Curry's 
Programming by Abstract Demonstration [Curry 78J. It was similar to 
Pygmalion, except that instead of dealing with concrete values. Curry manipu- 
lated abstract symbolic representations (e.g. "n") for data. While this made some 
things easier to program than in Pygmalion, it is interesting to note that in his 
thesis he reported making exactly the kinds of boundary errors mentioned above 
because he could not see actual values. 



We will illustrate the Pygmalion way of programming by defining the function 
"factorial." An ALGOL-Iike definition might be: 

integer function factorial (integer n) = 
if n = 1 then 1 
else n * factorial (n - 1) ; 

In Pygmalion, a person would define factorial by picking some number, say "6", 
and then working out the answer for factorial(6) using the display screen as a 
"blackboard." In the illustrations below, in order to save space I won't show the 
entire display screen for each "movie frame." Rather, I'll often "zoom in" on the 
parts of the screen that change from frame to frame. The programmer, however, 
always sees the entire screen: 



(Aside: the screen shots in this chapter are ft™ a recent HyperCard simulation 
of Pygmalion done on a Macintosh computer by Allen Cypher and myself 
Therefore the window appearance differs slightly from the actual system) 
The Pygmalion window consists of several parts: 

• "menu" area - a menu of icons and operations on icons (a rather mixed bag 
actually-today the icons would be in a palette and the operations in pull- 
down menus). The icons are under the "opcodes" and "control" headings 
and the operations are under the "icons" and "others" headings. This is the' 
complete icon editor. 

• "mouse value" area - Programmers could type a value here, and it would 
become "attached" to the mouse. It could then be deposited in icons. 



I" area - When the system is "remembering," the last couple of 
operations performed are displayed here. 



- Smalltalk expressions could be typed and evaluated here 
• "mouse" area - Pygmalion was implemented on a computer having a three 
button mouse; this area displayed the meaning of each button in the differ- 
ent modes. 

Except for the menu, these parts are not really important As I mention at the 
end, I would get rid of them all were I to reimplement the system today But 
with these as our resources, let's define factorial 

The entities with which one programs in Pygmalion are what I call "icons " An 
icon is a graphic entity that has meaning both as a visual image and as a machine 
object Icons control the execution of computer programs, because they have 
code and data associated with them, as well as their images on the screen. ITus 
(hstmgtusbes icons from, say, lines and rectangles in a drawing program, which 
have no such semantics. Pygmalion is the origin of the concept of icons as it 
now appears in graphical user interfaces on personal computers. After complet- 
ing my thesis, I joined Xerox's "Star" computer project The first thing I did was 
recast the programmer-oriented icons of Pygmalion into office-oriented ones 
representing documents, folders, file cabinets, mail boxes, telephones, wastebas- 
kets. etc. [Smith 82J. These icons have both data (e.g. document icons contain 
text) and behavior (e.g. when a document icon is dropped on a folder icon, the 
folder stores the document in the file system). This idea has subsequently been 
adopted by the entire personal computer and workstation industry 



To get started, I define an icon for the function: 



I create two sub-icons to hold the argument and value, and then draw some 
(crude) graphics to indicate the icon's purpose: 



Finally I make the outside border invisible: 

I now select the icon, invoke the "define" operation, and type in the name 
"factorial." This associates a name with the icon. The first thing the system does 
with a newly created function is to capture the screen state. Whenever the func- 
tion is invoked, it restores the screen to this state. This is so that the function will 
execute the same way regardless of what is on the screen when it is called. So 
before invoking "define," I make sure that I clear off any extraneous icons lying 
around. Icons may be deliberately left on the screen when a function is defined; 
these act as global variables. When the function is invoked, the "global" icons 
and their current values are restored to the screen (if they weren't already pre- 
sent) and are therefore available to the function. 

As soon as I invoke the "define" operation, the system enters "remember mode," 
causing every action I subsequendy do to be recorded in a script attached to the 
icon. So one defines functions simply by working out examples on the screen. 
Programs are created as a side effect. Here I'll arbitrarily work out the value of 
factorial for the number "6". (Choosing good examples is an art. One wants both 
typical values and boundary cases. Even in conventional programming, choosing 



test cases poorly is a principal sources of bugs. Pygmalion does not solve this 
problem.) 

I type -6" into the 



Whenever all the arguments to a function are filled in, the system immediately 
invokes it This was an experiment and is not inherent in programming by 
demonstration, or even necessarily a good idea. Functions in Pygmalion can be 
invoked even if their code is undefined or incomplete. When the system reaches 
a part of the function that has not yet been defined, it traps to the user asking 
what to do next. This was implemented by placing a "trap" operator at the end of 
every code script When a trap operator is executed, the system reenters "record 
mode." Every action subsequently performed is inserted in the code script in 
front of the trap. When the user invokes the "done" operation indicating that this 
code script is finished, the system leaves "record mode" and removes the trap 
operator. Otherwise the trap operator remains there. 

Factorial is now ready to execute, so the system invokes it. Since there is no 
code defined for it yet. it immediately traps to the programmer asking what to 
do. I move the factorial icon out of the way to make some room on the a 
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From here on, in order to save space in these pictures I won't show the window 
title or any area that is not directly involved in the actions being described. 

I want to test if the argument is equal to 1. Actually, I can see that it isn't 1; it's 
6. So why do I have to make a test? The answer is that Pygmalion is designed 
for programmers. Programmers know that functions will be called with different 
arguments, and they try to anticipate and handle those arguments appropriately. 
This is a little schizophrenic of Pygmalion: on the one hand it attempts to make 
programming accessible to a wider class of users; on the other, it relies on the 
kind of planning which only experienced programmers are good at But anticipa- 
tion is not inherent in programming by demonstration. Henry Lieberman's 
Tinker [Lieberman 80, 81, 82] doesn't force users to anticipate future arguments. 
In Tinker one writes code dealing only with the case at hand. When Tinker de- 
tects an argument that is not currently handled, it asks the programmer for a 
predicate to distinguish the new case from previous ones. The programmer then 
writes code to handle the new case. Tinker synthesizes a conditional expression 
from the predicate, the new code, and the existing code. Programmers never 
have to anticipate; they need only react to the current situation. This is an im- 
provement over Pygmalion's approach. 



At any rale, I've decided that I will need a conditional, so I invoke the T item 
in the menu. The system enters a mode waiting for me to specify where to put 
a icon. When I click the mouse, the icon is placed at the cursor lo- 
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^ — ► 



false 



The conditional icon consists of three sub-icons: one for the predicate and two to 
hold the code for the true and false branches. The predicate icon is the 
"argument" to the conditional; as soon as it has a value, the appropriate branch 
icon executes. 



To test if the factorial argument is equal to 1, 1 click on the menu item, point 
where I want the icon to go, and the system creates an equality-testing icon: 



It has two sub-icons to hold the data being tested. I drag the "6" down from fac- 
torial's argument icon and deposit it in the left hand sub-icon: 





• 




1 



(During playback— "watching the movie,"— Curry's system animated such 
value dragging, making for a quite articulate movie; this was an improvement 
over Pygmalion, which did no such animation.) Then I type a "1" into the right 
hand sub-icon: 




The equality icon is now fully instantiated, so it executes. It is defined to replace 
its contents with the result of the test Since 6 is not equal to 1, it replaces its 
contents with the symbol "false": 



false 



This value, like die value in any icon an the screen, can be used in other parts of 
the computation. In particular, it can be used in the conditional icon. I drag it 
into the conditional's predicate icon. The conditional icon is now fully instanti- 
ated, so it executes. The symbol "false" causes the false icon to be evaluated: 




raise ^ ^ true 



false 



But the false icon has no code denned for it yet, so it immediately traps back to 
me asking what to do. The false icon now becomes the "remembering" icon; 
previously it was the icon for factorial itself. So now I'm no longer defining 
code for the factorial icon; I'm defining code for the false branch of its condi- 
tional icon. In fact, from here on out all of the operations I perform will be at- 
tached to either the true or false branch of the conditional. 



Since I don't need the equality testing icon anymore, I'll get rid of it. I do this by 
invoking the "delete" menu item and then clicking on the equality icon. 



Now I want to compute n*f actorial (n-1 ) . I'll need a multiplication icon 
and a subtraction icon. These I get by invoking the appropriate menu items: 
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I drag factorial's argument (6) into the left hand icon in each of these. Then I 
type a 1 into the right hand icon in the subtracter 



The subtraction icon is now fully 



so it executes: 



Now I want to call factorial recursively on this value (5). I invoke the "call" 
menu item, type "factorial" when prompted, and click where I want it to go. The 
system creates a new instance of the factorial icon, the very one that I am in the 
middle of defining! As mentioned earlier, that factorial is not completely defined 
poses no problems for Pygmalion; it will execute what it has and ask for more 
when it runs out. The screen now looks like this: 
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I drag the 5 into factorial's argument icon. As usual since it is now fully instanti- 
ated, it executes. But now instead of being empty, there is quite a bit of code for 
factorial to execute: 



• It cleans off the screen and repositions the factorial icon. 

• It creates a conditional icon. 

• It creates an equality testing icon, fills it in, and evaluates iL Since 5 * 1, it 
evaluates to false again, 

• This causes the false branch of the conditional to be executed again. 

• This creates multiplication and subtraction icons, fills in the subtraction 
icon; and evaluates it, producing 4. 

• It creates yet another instance of the factorial icon, fills in the argument with 
4, and evaluates iL 
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And so on recursively, until finally factorial is called with 1 as the argument. At ^ scnen ■/«*" before factoriaU4) is 
this point, for the first time, the conditional's true icon is executed. But there is evaluated 
no code yet for this icon, so it immediately traps back to me asking for instruc- 
tions. The screen now looks like this: 
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In this case, we know exacUy what to do. Factorial 1) = 1. So I type 1 into the 
value icon: 



CD9=CD 



and invoke the "done" menu item. This returns from the last recursive call — fac- 
torial(l). It imme dia t ely traps asking for more instructions for the false branch of 
the conditional, since it was still trying to compute factorial(2) when the recur- 
sive call to factorial(l) was made. The screen now looks like this: 
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All that remains is to drag the value from the recursive call factorial(l) into the 
multiplication icon: 



This then is n*f actorial (n-1 ) , where n = 2. 1 drag this value into factori- 
al's value icon— the false branch as well as the true branch m 



The recursion now completely unwinds, since it knows how to do everything it 
needs to, and ends with the value of factorial(6). See the next page. Finally, in 
the picture on the following page, it restores the screen to its state when the 
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People sometimes ask me, if I were to build Pygmalion today, how would it be 
different? There are three main things I would do differently: 
1. The most important change I'd make is to address a broader class of users. 
Pygmalion was designed for a specific target audience: computer scientists. 
These people already know how to program, and they understand programming 
concepts such as variables and iteration. I would like to see if the approach could 
be applied to a wider audience: business people, homemakers teachers, children, 
the average "man on the street" This requires using objects and actions in the 
conceptual space of these users: instead of variables and arrays, use documents 
and folders, or cars and trucks, or dolls and doll houses, or food and spices. Or 
better yet, let users define their own objects and actions. Dan Halbeit captured 
this idea nicely with his concept of >ogramming in the language of the user in- 
terface" [Halbert 84]. 

2. The biggest weakness of Pygmalion, and of all the programming by demon- 
stration systems that have followed it to date, is that it is a toy system. Only 
simple algorithms could be programmed. Because of memory limitations I 
could execute factorial(3) but factorial^) ran out of memory. (Pygmalion was 
implemented on a 64 K byte computer with no virtual memory.) There was no 
way to write a compiler in it or a chess playing program or an accounting pro- 
gram, nor is it clear that its approach would even work for such large tasks a 
did. however, implement part of an operating system in it. at least on paper.) The 
biggest challenge for programming by demonstration efforts is to build a practi- 
cal system in which nontrivial programs can be written. 

3. 1 would put a greater emphasis on the user interface. Given what we know 
about graphical user interfaces today, it wouldn't be hard to improve the inter- 
face dramatically. I would put the icons into a floating palette as in HyperCard, 
replacing the long menu area that takes up so much screen space. I would put the 
operations on icons into pull-down menus. I would get rid of the bottom four ar- 
eas in the Pygmalion window entirely. The "mouse" area was necessary because 
the system was pretty modal; I would redesign it to be modeless. I would make 
greater use of overlapping windows (which hadn't been invented when this work 
was done). I would improve the graphics esthetics. With today's graphics re- 
sources-bit mapped screens, processor power, graphics editors, experienced bit 
map graphics artists— there is no longer any excuse for poor appearance Good 



esthetics are an important factor in the users' enjoyment of a system, and enjoy- 
ment is crucial to creativity. 

^^m^MM^^M^M Pygmalion was an early attempt to improve the process of progranuning. By 
studying how people think and communicate, I attempted to build a program- 
ming environment mat facilitated communication and stimulated people's ability 
to think creatively. While it never went beyond a toy system, Pygmalion embod- 
ied some ideas that still seem to have promise today: 

• It allowed ideas to be worked out via sketches on the screen and then was 
able to reexecute the sketches on new data. 

• It introduced icons as the basic entity for representing and controlling pro- 

• Programming was done by editing sketches and then recording the editing 
actions. This avoided the abstract step of writing down statements in a pro- 
gramming language. 

• Example data were always concrete, never abstract 

• It used analogical representations for data. This reduced the translation dis- 
tance between mental models and computer models of data. 

• It represented programs as movies. (Storyboards would probably be better, a 
la David Kurlander [Kuriander 88b].) 

To make it possible for the average person to write computer programs, I still 
believe that some combination of these ideas must be present. This is a worthy 
challenge for the 1990' s. 



Pygmalion 

How doe* the user create, execute and modify programs? 

The user creates programs by editing graphical snapshots of the computation. 
Essentially the user treats the display screen as an "electronic blackboard", using 
it to work out algorithms on specific concrete examples. 
Pygmalion provides graphical representations for the standard arithmetic, rela- 
tional, and Boolean operators. 

Partially specified programs could be executed. The system asks the user what to 
do next when it reaches the end of a branch. 

Programs can be modified, but then the rest of the modified branch must be re- 



Infereneing: No inferencing 

Types of examples: Multiple examples to demonstrate branches of a condi- 
tional, and recursion. 

Program constructs: Variables, loops, conditionals, n 
Machine, language, size, date: Smalltalk, 1975. 
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A Predictive Calculator 

Ian H. Witten 




One way of giving casual users access to some of the power of a computer with- 
out having to team formal programming methods is to enable tasks to be defined 
by giving examples of their execution, rather than by a procedural specification. 
This chapter summarizes a 1981 proposal for an electronic calculator that allows 
iterative computations to be inferred interactively from an initial part of the se- 
quence of key-presses [Witten 81]. The aim was to construct a device that is par- 
tially self-programming. The predictive calculator was implemented on a PDP- 
1 1/40 computer in 1981. It was intended as a demonstration rather than a practi- 
cal tool for casual users, and the user interface was not completely polished. 
The idea of defining problems operationally through examples was proposed in 
the "query-by-example" system for database retrieval [Zloof 77b], which was 
intended for a user with no programming and little mathematical experience. To 
specify what to retrieve, you present an example of an item that should be in- 
cluded. The scheme illustrates that useful interactive systems can be built that 
allow users to define non-trivial tasks by example. One of its advantages is that 
it frees users from thinking of the retrieval problem in an artificially sequential 
form: instead they can specify the links, conditions, and constraints in any order. 



Figure 1. Trace of a sorting process 



t:=a(0) s:=0 i:=l i< t -i a(i*l)<a(i) s:=i x:=a(i) 
a(l):=a(i*l) a(i + l) : = x i:=i + l i<t -l a(i*l)>a(i) 
Kt-1 a,i + l,<a(i) s:=i x:=a(i, a(i,: = a<i+l, aU + l, : =x 

i<t-l a|i + l)<a(i) S: =i x:=a(i) a(i):=a(i+l) 
a(i*l) : =x i =t -l s >0 s:=0 i:=l i< t -l a (i*l)>a(i) 

i<t-l a(l + l)«|i) = i<t -l a{i + l)<a(i) s:=i 
x:=a(l) a(i):=a<i*l) a(i*l):» X i:=i+l i< t -i 
a(i+l)>a(i) i-.=Ui i =t -l s>0 s:=0 i.=i i<t-i 
a(i*l)>a(i) i:=i + i i<t -i a(iU)<a|i) s:=i x:=a(i) 
a(i) : =a{i*l) a(i*l) !=x 1:=i+1 1<t j a (i+l)>a(i) 
i:=i+l i<t-l a(i*l)>a(i> i==i + l i=t-l s>0 s-=0 i-l 
i<t-l a(i*l)>a(i) i:=Ui i<t-i a(Ul)>a(i) i:=i + i 
i<t-l a(i + i)>a(i) i:= i+ i i^-x a(i*l)>a(i) i:=jUl 
i=C-l s=0 end 



t:a(0). 
a(l) ... 



number of elements to be sorted, minus 1 
a(t-i) elements to be 



In contrast, the present work explores bow strictly sequential information can be 
inferred by a machine from an initial subsequence. 



t of programs from traces is another domain in which the 
problem is specified by an example. When a detailed trace of a particular execu- 
tion of a program is available at the right level of generality, program inference 
can be easy. For instance, [Biermann 72] discusses the inference of luring ma- 
chines from traces of sample computations. The problem is more usefully ad- 
dressed at a higher level than Turing-machine instructions, and [Gaines 76] gave 
the programming-language-level trace in Figure 1 as an example of a sorting 
program that performs a simple bubble-sort. The flowchart in Figure 2 can be 
derived from the trace simply by assigning different states to different state- 
ments and drawing sequential links between them. It is correct and complete ex- 
cept for a link from the assignment "i : =i" to the test "i=t-i" which was not 
demonstrated by that particular example. However, it is unlikely that such a 
trace would be entered without error, and even if it were, it may not be any eas- 
ier for a casual user to generate than a structural description-a program-for 
the sorting process. Traces can be useful, though, in more limited domains-as 
this chapter will show. 



f t:.a(0) 



Figure 2. Program formed from the 
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When using interactive computers, there are many situations in which it is hard 
to decide whether to do a minor, but repetitive, task by band or write a program 
to accomplish it. Simple, repetitive arithmetic calculations often present this 
quandary. For example, to plot y = xe l ~ x for a dozen or so values of x, should 
one use a hand calculator or write a BASIC program? The first is easier and 
more certain; it will not take more than 10 minutes. The second may be quicker, 
but could require consulting the manual to refresh one's memory of the vagaries 
of BASIC syntax. 

The Predictive Calculator is a system that quietly "looks over the shoulder" of 
the user, forming a model of the action sequence. After a while it may be able to 
predict what keys will be pressed next Any that cannot be predicted correspond 
to "input" Thus the system can eventually behave exactly as though it had been 
explicitly programmed for the task, waiting for the user to enter a number and 




simulating the appropriate sequence of key-presses to calculate the answer. The 
user must pay a price for the service, for the system cannot help but be wrong 
occasionally: moreover, extra keys must be provided to enable predictions to be 
accepted or rejected 

Sample dialogues 

The device is based on the simple, non-programmable pocket calculator shown 
at the beginning of this chapter. An adaptive model is constructed of the se- 
quence of keys that are pressed. If the task is repetitive (like computing a simple 
function for various argument values), the system will soon catch on to the se- 
quence and begin to activate the keys itself. When the prediction is wrong an 
undo key was envisaged to enable the user to correct it by undoing a single step 
of the sequence (though, as noted above, the interface was never completely fin- 
ished). In order to keep control in the hands of the user, predictions are indicated 
but not actually made until they have been confirmed by an accept key. One can 
imagine being able to switch into a "free-run" mode after having gained enough 
confidence in the predictions, but this was not implemented. 

Figures 3-5 give examples of this "self-prograniming" calculator. The first 
shows the evaluation of xel~* for a range of values of x. The keys pressed by the 
operator are in normal type; those predicted by the system are shaded. From 
halfway through the second iteration onwards, the device behaves as though it 
had been explicitly programmed for the job (except for the need to repeatedly 
press accept or switch into free-run mode). It waits for (be user to enter a num- 
ber, and displays the answer. It takes slightly longer for the constant 1 to be 
predicted than the preceding operators because the system requires an additional 
confirmation before venturing to predict a number (see below). Provided the 
user enters variable data before any fixed constants (and people generally do 
this), there is no difficulty in getting the calculator to pause when the answer has 
been reached; it stops automatically whenever it cannot predict the next element 
in a sequence and this occurs at the end of each of the four calculations in 
Figure 3. 

Figure 4 shows the evaluation of 



81og2 

for various values of x. The first line stores the constant log 2 in memory. 




Figure 3. Using the self-programming 
uate xe 1 -* for wri- 
ts etfx 




Figure 4. Evaluating 

1+ log x / Stag! for various values ofx 
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x 10 » (answer » -2.69856) 



Figure 5. Evaluating a more complex 



500/ 4000 x 180 i cos x 2 x .9 mcm*i .9xx=* 1 -mr= togx 10 ♦ 20 ♦ 2.69858 * 
1<j00>40dW«(> = &'*2 XJim^^J^v^X- mr = tog x 10 ♦ 20 ♦ 2.6985B . 
%SM$m-Win*jcoti& x .9 me m*. 'S X X a Vi- mr » tog x 10 + 20 + 2.69658 . 
2000V 4TOx480 = coi'x 2 x &mem**axi =♦ 1 - mr =, tog x 10 ♦ 20 + 2.69858 = 
2500yv^'xl80o 00s x 2 x j9 mc'mW.9 xx - *1- mr = tog x 10 ♦ 20 ♦ 2.69856 = 
3000 / 4000 x 180 = cos x 2 x 3 mc m+= .9 x x = ♦ 1 - mr = tog x 1 0 + 20 ♦ 2.69858 = 
3500 / 40iDO x 180 - eos x 2 x a mc m+= a x x - *1 1 - mr » tog x 10 ♦ 20 ♦ 2.69858 - 
4000/*DO0 x18d'»cosx 2xSmcim= .9 X x = *1-mr = bgx10 + 20 + 2.69858 = 



More complicated is the evaluation of 

20* lOlos [l ♦ P-2.« jgg] -10 log [1 +<fi-2a«»45) 

for a=0.9, shown in Figure 5. Since the calculator possesses only one memory 
location, it is expedient to compute the last subexpression first and jot down the 
result The result of this calculation (-2.69858) only has to be keyed twice be- 
fore the system picks it up as predictable. Some interference occurs between this 
nubal task and the main repeated calculation, and three suggestions had to be 
un d o n e " by the user. 



If an erroneous suggestion is made, indicated by an "undo." the system is cau- 
tious about making a prediction in that context in the future. Any step that is un- 
done is erased from the system's model of the interaction, leaving no record of 
its existence. Of course, the user will (presumably) have followed the undo by 
an action that is different from the one predicted, creating a conflict in the 
model. Consequently when the same context occurs again, no prediction will be 
offered to the user. However, the system will internally monitor its own predic- 
tions in that context, and when several correct predictions have been made sub- 
sequently (but not presented to the user), it will eventually venture its sugges- 
tions. This is the reason why the penultimate «+" on each line of Figure 5 has to 
be keyed by the user several times. Both of the sequences 

log x io = 

log x 10 + 

had occurred in the interaction, but a long run of the second was enough to — 
it to be used e — " a 



As Figure 5 implies, any number that appears in the input sequence is identified 
and treated as a single unit To enable the predictions that follow numbers to be 
made in time for them to be useful, it was necessary to provide a special key 
with which each number is terminated. Of course, operations like "cos" al- 
though written as three letters in the figure, already correspond to a single 
keystroke and so no terminator is -- J " 



The modeling technique 

The system uses an extremely simple method to form its predictions The user's 
input sequence is characterized by the set of overlapping k-tuples of symbols 



that occur in it, for some limited context length k (k=4 in the examples of 
Figures 3-5). The symbols are either numbers or operation keys. For instance. 
Figure 6 shows the 4-tuples that are stored for the sequence of Figure 3. 

The first row shows the first four symbols in the sequence, the second shows the 
4- tuple starting at the second symbol, the third starts at the third symbol, and so 
on. To use these for prediction, whenever the last k-l symbols entered match 
those that begin a stored tuple, the last symbol of that tuple is predicted. For in- 
stance, if the user enters "m+= +/- +** this matches the third row, so the system 
will predict 1. 

This modeling technique, known as a "length-k" modeling [Witten 79], arose 
out of context modeling techniques [Andreae 77]. It turns out that the k- tuples 
can be stored economically by massaging them into the form of an automaton 
model, although this is hardly necessary with modern store sizes. A more power- 
ful, though slightly less space-efficient, prediction technique is to use partial 
matching on the k-tuples in conjunction with a trie-structured model (e.g. the 
REACTIVE KEYBOARD [Darragh 92] and the PPM text compression method 
[Bell 90]); that would be the natural way to implement the predictive calculator 
nowadays. What is surprising is that the simplistic modeling technique that was 
actually implemented performed so well in practice. 

A modification to the modeling technique was made to suit the calculator situa- 
tion. It is clear from a cursory analysis of calculator sequences that numbers and 
operators should be treated differently, for a typical sequence comprises differ- 
ent numbers embedded in a fixed template of operators. This rule is not univer- 
sal, because fixed constants appear in the stream as weU as variable input data. 
Note how the constants 1 in Figure 3, 8 and 1 in Figure 4, and 4000, 180, 2, .9, 
1, 10, 20 and 2.69858 in Figure 5 are all quickly picked out as predictable by the 
system. 

In order to prevent differences in data values from rendering the length-k se- 
quences inoperative, two parallel sets of k-tuples were used. One was exactly as 
shown above, whereas the other mapped all input numbers into the same token 
<num>, yielding Figure 7 from Figure 6. 

Predictions from this model were only used when the other one failed to yield a 
prediction. For example, when the first prediction of Figure 3 is made, the "+ / -" 
on the second line, it is predicted from the tuple "<num> mc nn-= +/-", because 




Figure 6. The basic model that is stored 
for the sequence of Figure 3 
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Figure 7. The mopped model that u 
constructed from Figure 6 



the actual context ".2 mc nu- has not been seen before. Funhennorc, the sys- 
tem was consulted to be mo re conservative about predicting numbers than 
operators. No prediction was made unless it would have been correct the previ- 
ous* i times ,r occurred, and n was set differentiy for operator, (n=l) and num- 
be^«=2). Thus the tuples +/ - + 1" is not used to predict the 1 on the 
second line of Figure 3, but it is used on the third line. 
Except for its ability to parse numbers in the symbol string and to distinguish 
numbers from operators, the system incorporates no knowledge about the calcu- 
tator or calculation sequences. For example, it will learn nonsense sequences 
mm l l + ♦ 1 1 ♦ ♦» and regurgitate them just as readily as syntactically 
meaningful sequences. (The sequence "xx» that occurs in Figure 5 may seem 
anomalous; in fact it is the calculator's way of squaring a number.) The proce- 
dure ,s entirely lexical and does not recognize patterns of numbers. For exa^ 
u. the sequence .1, .2, .3. ... of Hgure 3 it is evident what should come next but 
the system does not spot the pattern. Also, it does not notice multiple occur- 
rences of die same input data. For example, if x exp(l-x) were to be evaluated 

r^T" With ° Ut 3 X W0Uld tave «° * ««» ^ mode.- 

»ng procedure is incapable of exploiting this redundancy. 

A common source of variation in calculator sequences is the discovery of an 
easier way to do the task. For example, halfway through Figure 5 one may de- 
cide to enter 1.81 directly instead of xx . + ITus is unlikely tie 
Present example, for no penalty is associated with the latter sequence once it has 
been learned. However, if the simplifying discovery is made early on in the in 
teracuon it may cause s< — 



Th^system illustrates how systematic processing of a "history list" of previous 
mteracdons can be used to predict future entries. Rather than being invoked ex- 
plicitly, die history list is accessed implicitly to provide assistance to the user. 
Despite the fact mat the modeler has no knowledge of the syntax or semantics of 

reL^r 6 ' k! 15 KB t T blV SUaXSSfUl FW eXam P ,e ' when three short 
repeuuve problems of Figure 3-5 were concatenated into a single input se- 
quence, over 75% of the sequence elements were predicted and less than 1.5% 
of predicuons were incorrect There is always a temptation to adjust system pa- 
rameters ,n retrospect to yield good performance, and when this was done 



nearly 80% of correct predictions were achieved with an error rate of under 
0.5%. 

The basic idea behind the predictive calculator has been engineered into a device 
called the REACTIVE KEYBOARD that accelerates typewritten communication 
with a computer system by predicting what the user is going to type next 
[Darragh 92]. Obviously, as with the calculator, predictions are not always cor- 
rect, but they are correct often enough to form the basis of a useful communica- 
tion device. Since predictions are created adaptively, based on what the user has 
already typed in this session or in previous ones, the system conforms to what- 
ever kind of text is being entered. It has proven extremely useful for people with 
certain kinds of communication disability. 



It is a great pleasure to acknowledge the influence and encouragement of Brian Aclaiowledgments .-. j * ! 
Gaines, John Cleary, and especially John Andreae. John Darragh skillfully im- 
plemented the predictive calculator program on a PDP-1 1/40 computer. 



Predictive Calculator 

Application domain: Calculator 

Ho* does the user create, execute and modify programs? 

User creates programs simply by using the calculator. There is no special learn- 
Hie only user interaction, beyond doing the task, is to press Undo 



By rejecting a prediction, the user forces the calculator to learn a n 



Keeps a history of all four-token sequences (a token is a number or an operator) 
Whenever the user types three tokens that match a recorded sequence, the 
Predictive Calculator predicts the fourth. (Any fixed value can be used instead of 
"four"). 



Can ask fox user input for any data that varies, but cannot create a variable (e a 
it does not identify the two X's in "X2 + 5X") 

Type* end source* of information: 

No syntactic rules. (e.g. it will happily learn the sequence "1 1 + + 1 1 + + " if 
that is what the user enters) 

Machine, language, size, date: PDP-11/40, 1982. 



